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The  Effects  of  Knowledge  of  Results  and  Test  Difficulty  on 
Ability  Test  Performance  and  Psychological  Reactions  to  Testing 


Sufficient  Biotivation  is  necessary  in  order  to  adequately  measure  an 
individual's  ability  (e.g. , Cronbach,  1970;  Samuda,  1975;  Terman,  1916).  The 
provision  of  immediate  feedback  or  "knowledge  of  results"  (KR)  during  testing 
is  one  possible  method  for  increasing  or  maintaining  high  levels  of  motivation 
(Bayroff,  196A;  Betz,  1975;  Betz  & Weiss,  1976b;  Ferguson  & Hsu,  1971;  Strang 
& Rust,  1973;  Zontine,  Richards,  & Strang,  1972).  On-line  computerized 
testing  has  made  the  provision  of  KR  a relatively  simple  matter;  the  ease 
with  which  KR  can  be  administered  is  an  added  advantage  of  computerized  adap- 
tive testing,  which  lies  beyond  the  purely  psychometric  benefits  of  such 
procedures . 

To  study  the  effects  and  possible  benefits  of  computer-administered  KR, 
Betz  and  Weiss  (1976a,  1976b)  administered  multiple-choice  tests  of  verbal 
ability  to  college  undergraduates  at  the  University  of  Minnesota;  the  tests 
were  administered  either  with  or  without  KR  after  each  item  response.  Their 
data  showed  higher  testee  performance,  as  measured  by  maximum  likelihood 
ability  estimates,  for  all  students  in  the  KR  condition  and  significantly 
higher  performance  levels  for  low-ability  students.  Perceptions  of  test 
difficulty  were  more  accurate  for  students  receiving  KR;  these  students  also 
exhibited  higher  levels  of  motivation,  as  assessed  by  post-test  measurements. 
The  data  also  indicated  that  students'  reactions  to  the  provision  of  KR  became 
more  favorable  as  the  proportion  of  positive  feedback  increased.  Because  KR 
Increased  performance  and  motivation  in  the  Betz  and  Weiss  studies  and  because 
individuals  reacted  more  favorably  to  the  provision  of  KR  as  the  proportion  of 
positive  feedback  increased,  an  analysis  of  the  joint  effects  of  provision  of 
KR  and  the  proportion  of  positive  feedback  (test  difficulty)  was  Initiated. 


Method 


Pvaaedure 


Subjects.  Participating  in  this  study  were  561  undergraduate  students 
enrolled  in  an  Introductory  psychology  course  at  the  University  of  Minnesota 
in  the  fall  of  1975.  All  students  were  volunteers  who  received  points  towards 
their  final  course  grade  for  participation  in  the  experiment.  Students  were 
sequentially  assigned  to  experimental  conditions. 


Test  administration.  All  students  were  tested  at  individual  cathode-ray 
terminals  (CPvTs)  connected  to  a Hewlett-Packard  9600E  Real-Time  computer  system. 
Instructional  screens  explaining  the  operation  of  the  CRTs  preceded  the  actual 
testing  and  were  similar  to  those  described  in  DeWltt  and  Weiss  (1974,  pp.  36-53). 
In  addition,  students  were  Informed  that  they  would  have  as  much  time  as  they 
needed  to  finish  the  test.  A proctor  was  present  in  the  testing  room  to 
provide  assistance  in  the  operation  of  the  equipment. 
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Each  test  consisted  of  50  five-alternative  multiple-choice  items.  During 
the  test,  items  were  presented  on  the  CRT  screen  and  students  responded  by 
typing  in  a number  corresponding  to  the  chosen  alternative  for  each  item. 
Follov/ing  the  test,  the  students  were  asked  to  indicate  their  feelings  about  the 
test  by  responding  to  a series  of  questions  designed  to  assess  their  psycho- 
logical reactions  to  the  test.  Students  not  provided  KR  were  asked  18  questions, 
while  those  receiving  KR  were  asked  the  same  18  questions  plus  an  additional 
8 questions  concerning  their  reactions  to  the  provision  of  KR. 

Design 

Tnrhpendent  vapiahles.  A three-way  factorial  design  was  employed  in  the 
study.  One  factor  was  immediate  knowledge  of  results  (KR).  Students  in  the 
KR  condition  were  informed  by  the  computer  immediately  after  their  response  to 
a question  whether  the  reponse  was  correct  or  incorrect.  After  an  incorrect 
response,  they  were  told  which  of  the  alternatives  was  correct.  Students  in  the 
no-KR  condition  received  no  feedback.  Ability-test  strategy  was  another  factor. 
Individuals  received  either  a conventional  peaked  ability  test  or  a fixed-length 
stradaptlve  ability  test  (Weiss,  1973).  A third  factor  was  test  difficulty  or 
proportion  of  positive  feedback.  Three  conventional  tests  and  three  stradaptlve 
test-administration  procedures  were  designed  such  that  students,  on  the  average, 
would  answer  approximately  40%,  60%,  or  80%  of  the  test  questions  correctly. 

Level  of  difficulty  (high,  medium,  or  low)  was  Inversely  related  to  the  propor- 
tion of  positive  feedback  an  individual  received,  whether  that  feedback  was 
explicit  as  in  the  KR  condition  or  subjective  as  in  the  no-KR  condition. 

Dependent  vaeiablee.  Both  the  ability-test  performance  and  the  psycholog- 
ical reactions  of  the  testees  were  of  Interest.  Performance  was  measured  by 
maximum  likelihood  ability  estimates  computed  for  each  individual  by  solving 
the  likelihood  equation  for  Blrnbaum's  (1963,  p.  459)  three-parameter  logistic 
model.  Proportion-correct  scores  were  also  computed  in  order  to  assure  that 
the  tests  were  of  appropriate  difficulty.  Proportion  correct,  however,  was 
not  used  as  an  ability  measure  per  se,  since  the  stradaptlve  test-administration 
procedure  was  designed  to  yield  an  arbitrary  proportion  correct  for  each 
student  and  since  differences  in  proportion-correct  scores  between  students 
administered  the  three  different  conventional  tests  were  determined  to  a degree 
by  the  construction  of  those  tests. 

The  reactions  of  individuals  to  the  tests  were  determined  using  the 
responses  of  the  students  to  psychological  reactions  items  administered  following 
the  test.  The  four  scales  constructed  from  these  items  measured  students' 
perceptions  of  the  test's  difficulty;  their  level  of  anxiety  during  testing; 
their  motivation  to  do  well  on  the  test;  and,  for  testees  in  the  KR  conditions, 
their  reactions  to  the  provision  of  KR.  The  items  in  the  Difficulty-Perception, 
Anxiety,  Motivation,  and  KR-Reactlon  scales  are  shown  in  Appendix  Tables  A 
through  D,  respectively.  Also  shown  are  the  serial  positions  of  the  items  in 
the  post-test  battery  of  items.  Response  options  on  the  Difficulty-Perception, 
Anxiety,  and  Motivation  scales  were  weighted  on  a scale  from  1 to  5,  while  those 
on  the  KR-Reactlon  scale  were  weighted  on  a scale  from  1 to  't.  These  weights 
are  also  shown  in  the  appendix  tables.  The  scales  were  scored  so  that  increasing 
scale  scores  corresponded  to  increasing  anxiety,  Increasing  motivation, 
perceptions  of  increasing  difficulty,  or  increasingly  positive  reactions  to  KR. 
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Teet  Conatvuation 

Item  pool.  The  item  pool  from  which  the  tests  were  constructed  consisted 
of  569  five-alternative  multiple-choice  vocabulary  Items.  Each  Item  had 
associated  with  it  a normal-ogive  discrimination  (a)  and  difficulty  (b) 
parameter  estimate  (Lord  & Novlck,  1968).  These  parameters  were  based 
on  data  derived  from  samples  of  University  of  Minnesota  undergraduate  students. 
The  norming  sample  and  the  original  norming  procedures  are  described  in  McBride 
and  Weiss  (1974).  Each  of  the  five-alternative  items  was  assumed  to  have  a 
guessing  parameter  (a)  of  .20. 

Conventional  teats.  Items  were  chosen  for  the  three  50-ltem  peaked  conven- 
tional tests  so  that  the  mean  normal-ogive  discriminations  would  be  equal 
(approximately  a=.80)  and  the  mean  normal-ogive  difficulties  would  be  approxi- 
mately b»-2.00,  -.35,  and  1.30  for  the  low-,  medium-,  and  hlgh-dlf f iculty  tests, 
respectively. 

Stradaptive  testa.  In  order  to  form  a stratified  Item  pool  for  adaptive 
testing  (VJeiss,  1973),  items  ranging  in  difficulty  from b—3.0  to  b»+3.0  were 
grouped  into  nine  non-overlapping  strata  on  the  basis  of  their  difficulty- 
parameter  estimates.  The  difference  between  the  lowest  and  the  highest 
difficulty  in  a stratum  was  constant  at  b“.67.  Within  a stratum,  items  were 
arranged  in  descending  order  of  their  discrimination-parameter  estimates.  A 
stratum  contained  at  most  30  items,  and  no  items  were  included  with  discrimina- 
tion-parameter estimates  of  less  than  .30. 


In  a stradaptive  test,  branching  between  items  is  usually  determined  by 
whether  an  individual's  response  to  the  immediately  preceding  item  was  correct 
or  incorrect  (Weiss,  1973).  The  stradaptive  administration  procedure  was 
modified  in  this  study  so  that  branching  to  a more  (or  less)  difficult  item 
was  based  on  whether  the  individual's  overall  proportion  correct  was  over 
(or  under)  a target  value.  In  order  to  achieve  average  proportion-correct 
scores  of  .40,  .60, and  .80  for  the  low-,  medium-,  and  hlgh-diff iculty  tests, 
target  values  of  .36,  .60, and  .84  were  employed  for  the  three  tests,  respectively. 
The  target  values  were  determined  from  a prior  analysis  of  simulated  responding. 
Branching  was  terminated  after  the  administration  of  50  items. 
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Revised  parameter  eetimatee.  During  the  interval  between  the  construction 
of  the  tests  and  the  analysis  of  these  data,  the  item-parameter  estimates  were 
revised  according  to  a procedure  outlined  in  Prestwood  and  V/eiss  (1977,  Appen- 
dix A).  The  parameter  estimates  for  items  in  the  three  conventional  tests  are 
shown  in  Appendix  Table  E.  The  mean  b's  for  the  low-,  medium-,  and  high-diffi- 
culty  tests  were  -2.32,  -.70,  and  .68,  respectively.  The  mean  a's  for  the 
low-,  medium-,  and  high-difficulty  tests  were  .65,  1.07,  and  1.76.  An 
inspection  of  the  revised  item  parameters  for  the  items  in  the  stratified 
item  pool  designed  for  stradaptive  tests  (Appendix  Table  F)  shows  some  minor 
overlap  in  difficulties  between  adjacent  strata  and  some  departure  from  the 
descending  order  of  discriminations  within  a stratum.  The  stratified  pool, 
however,  generally  corresponds  very  closely  to  the  desired  structure  specified 
for  a stradaptive  test. 

Data  Analueis 

Proportion- aorr eat  measures.  The  proportion  of  items  correctly  answered  was 
calculated  for  each  student.  The  mean  and  standard  deviation  of  these  measures 
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were  then  calculated  for  each  of  the  12  experimental  conditions  In  order  to  assess 
the  accuracy  with  which  the  conventional  and  stradaptlve  tests  yielded  the  desired 
levels  of  test  difficulty. 

Ability  estimates.  A maximum- likelihood  estimate  of  ability  was  calculated 
for  each  Individual, using  the  item  parameters  shown  in  Appendix  Tables  E and  F. 
The  mean  and  standard  deviation  of  these  estimates  were  calculated  for  each 
experimental  condition  and  each  combination  of  conditions.  A classical  three-way 
factorial  analysis  of  variance  was  then  performed  on  the  data  in  order  to  assess 
the  effects  of  KR-provislon , test  strategy,  and  test  difficulty  on  mean  student 
performance. 

Peyahologioal  reactiona.  The  means  and  standard  deviations  of  scores  on  the 
four  psychological  reactions  scales  were  calculated  for  all  experimental  treat- 
ments and  for  all  combinations  of  treatments.  The  effects  of  KR-provision , test 
strategy,  and  test  difficulty  on  mean  Difficulty-Perception,  Anxiety,  and  Motiva- 
tion scores  were  assessed  by  three-way  factorial  Analyses  of  variance.  A two-way 
factorial  analysis  of  variance  was  performed  on  data  from  the  KR-Reactlon  scale. 
This  analysis  investigated  the  effects  of  test  strategy  and  test  difficulty  on 
individuals'  reactions  to  KR  for  those  students  in  the  KR  conditions.  Relative 
frequencies  of  response  option  endorsements  were  also  calculated  for  items  on 
the  KR-Reactlon  scale. 

Cronbach's  a reliability  coefficient  was  calculated  for  each  of  the  psycho- 
logical reactions  scales.  In  additlo7>,  the  inter-scale  Pearson  product-moment 
correlation  coefficients  were  computed. 

Results 

Ability  Test  Data 

Proportion  aorreat.  Table  1 shows  the  means  and  standard  deviations  of  the 
proportion-correct  measures  for  each  experimental  condition.  Inspection  of  the 

Table  1 


Means  and  Standard  Deviations  of  Proportion  Correct  for  Conventional 
and  Stradaptlve  Tests  With  and  Without  KR  at  Three  Levels  of  Difficulty 


Experimental  Condition 

Experimental 

Test 

KR 

No-KR 

Condition 

Difficulty 

N 

Mean  S . D . 

N Mean  S.D. 

Conventional  Test 

Low  Difficulty 

.800 

48 

.783 

.106 

45 

.808 

.103 

Medium  Difficulty 

.600 

47 

.608 

.147 

49 

.592 

.141 

High  Difficulty 

.400 

46 

.451 

.188 

46 

.364 

.153 

Stradaptlve  Test 

Low  Difficulty 

.800 

44 

.828 

.064 

45 

.824 

.046 

Medium  Difficulty 

.600 

49 

.617 

.031 

47 

.610 

.041 

High  Difficulty 

.400 

49 

.434 

.103 

46 

.417 

.076 
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table  reveals  that  tests  in  each  condition  achieved  the  appropriate  target 
proportion  correct  with  a good  degree  of  accuracy.  The  largest  discrepancy 
(.051)  was  between  the  t.irget  value  of  .400  and  the  actual  value  of  .451  for 
the  low-difficulty  conventional  test  administered  with  KR. 


Table  2 


Means  and  Standard  Deviations  of  Maximum  Likelihood  Ability  Lstimates  for 
Conventional  and  Stradaptive  Tests  With  and  Without  KR  at 
Three  Levels  of  Difficulty,  and  Three-Way  ANOVA  Results 


Experimental 

Condition 

Experimental 

KR 

Condition 

No-KR 

Combined 

Conditions 

N Mean  S.D. 

N Mean 

S.D, 

N 

Mean 

S.D. 

Conventional  Test 

Low  Difficulty 

48  -. 

50  .98 

45  -.21 

1.00 

93 

-.36 

.99 

Medium  Difficulty 

47  -. 

38  1.15 

49  -.44 

,96 

96 

-.41 

1.06 

High  Difficulty 

46  -. 

21  1.28 

46  -.74 

1.23 

92 

-.48 

1.28 

Str.idaptlve  Test 

Low  Difficulty 

44  -. 

24  1.05 

45  -.10 

,93 

89 

-.17 

.99 

Medium  Difficulty 

49  -. 

26  .81 

47  -.32 

.99 

96 

-.29 

.90 

High  Difficulty 

49  -. 

20  1,13 

46  -.31 

.98 

95 

-.26 

1.05 

Combined  Conditions 

Conventional  Test 

141  -. 

37  1.14 

140  -.46 

1.08 

281 

-.42 

1.11 

Stradaptive  Test 

142  -. 

23  .99 

138  -.25 

.97 

280 

-.24 

.98 

Low  Difficulty 

92  -, 

38  1.02 

90  -.16 

.96 

182 

-.27 

,99 

Medium  Difficulty 

96  -. 

32  .99 

96  -.38 

.97 

192 

-.35 

.98 

High  Difficulty 

95  -. 

21  1.20 

92  -.52 

1.13 

187 

-.36 

1.17 

Total 

283  -. 

30  1.07 

278  -.36 

1.03 

561 

-.33 

1.05 

Three-Way  Analysis 

of  Variance 

Source  of 

Sum  of 

Mean 

Variation 

Squares 

DF  ■ 

Square 

F 

t 

Main  Effects 

KR 

.41 

1 

.41 

.38 

.999 

Test  Strategy 

4.36 

1 

4.36 

3.98 

,044 

Difficulty  Level 

1.05 

2 

.52 

.48 

.999 

Two-Way  Interactions 

KR  X Test  Strategy 

.28 

1 

.28 

.25 

,999 

KR  X Difficulty  Level 

6.36 

2 

3,18 

2.91 

,054 

Test  Strategy  x Difficulty 

Level 

.24 

2 

.12 

.11 

.999 

Three-Way  Interaction 

KR  X Test  Strategy  x 

Difficulty  Level 

2.02 

2 

1.01 

.92 

.999 

Residual 

601.18 

549 

1.10 

Total 

615.83 

560 

1.10 

*Estlmated  probability  of  error  in  rejecting  null  hypothesis. 


Maxiimwi  likelihood  abilitu  eatimatos.  Table  2 shows  the  means  and  standard 
deviations  of  the  maximum  likelihood  ability  estimates  as  a function  of  KR 
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condition,  test  strategy,  and  test  difficulty.  The  results  of  the  three-way 
analysis  of  variance  are  also  shown.  The  analysis  of  variance  revealed  a 
significant  (p<.05)  main  effect  for  test  strategy.  The  mean  ability  estimate 
for  students  administered  the  stradaptlve  tests  (-.2A)  was  significantly 
higher  than  the  mean  for  students  taking  the  conventional  tests  (-.42). 

There  was  also  a marginally  significant  (p=.054)  two-way  interaction 
between  the  provision  of  KR  and  test-difficulty  factors.  This  Interaction  is 
shown  graphically  in  Figure  1.  Interestingly,  Figure  1 shows  that  the 
effects  of  test  difficulty  on  test  performance  were  opposite  in  direction, 
depending  on  whether  or  not  KR  was  provided.  When  KR  was  provided,  the  mean 
student  ability  estimate  was  highest  on  the  most-difficult  tests  (-.21)  and 
lowest  on  the  least-difficult  tests  (-.38).  The  mean  ability  estimate  for 
students  in  the  no-KR  conditions  was  highest  on  the  least-difficult  tests 
(-.16)  and  lowest  on  the  most-difficult  tests  (-.52). 


Figure  1 

Mean  Maximum-Likelihood 
Ability  Estimates  as  a 
Function  of  KR-Provision 
and  Test  Difficulty 
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PeuahologiQdl  Regationa  to  Testing 

Diffiauttu  Pepgeption,  The  means  and  standard  deviations  of  scores  on 
the  Difficulty-Perception  scale  are  shown  in  Table  3 along  with  the  results 
of  a three-way  analysis  of  variance  of  these  data.  The  expected  main  effect 
due  to  test  difficulty  was  highly  significant  (p<.001).  In  addition,  the 
main  effect  of  KR-provlslon  was  also  significant  (p<.01).  Students  receiving 
KR  perceived  the  tests  as  being  less  difficult  than  did  those  not  receiving 
KR.  The  mean  Difficulty-Perception  score  for  students  in  the  KR  conditions 
was  -.94,  while  the  mean  for  students  In  the  no-KR  conditions  was  -.13. 
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Table  3 

Means  and  Standr  i Deviations  of  Difficulty-Perception  Scores 
for  Conventional  and  Stradaptive  Tests  With  and  Without  KR 
at  Three  Levels  of  Difficulty,  and  Three-Way  ANOVA  Results 


Experimental 

— 

Ext 

KR 

)erlmental 

Cond it  ion 

No-KR 

Combined 

Condition 

IS 

Condit ion 

N 

Mean 

S.D. 

N 

Mean 

S.D. 

// 

Mean 

S.D. 

Conventional  Test 

Low  Difficulty 

48 

-3.59 

3.08 

45 

-2.91 

3.24 

93 

-3.26 

3.16 

Medium  Difficulty 

47 

-1.13 

3.53 

49 

-.59 

3.05 

96 

-.85 

3.29 

High  Difficulty 

46 

1.84 

4.82 

46 

3.69 

4.44 

92 

2.76 

4.70 

Stradaptive  Test 

Low  Difficulty 

44 

-4.98 

2.46 

45 

-3.27 

2.82 

89 

-4.12 

2.77 

Medium  Difficulty 

49 

-.79 

2.35 

47 

-.17 

3.00 

96 

-4.9 

2.69 

High  Difficulty 

49 

2.69 

3.37 

46 

2.34 

3.39 

95 

2.52 

3.37 

Combined  Conditions 

Conventional  Test 

141 

-1.00 

4.44 

140 

.07 

4.50 

281 

-.46 

4.49 

Stradaptive  Test 

142 

-.89 

4.15 

138 

-.34 

3.82 

280 

-.62 

4.00 

Low  Difficulty 

92 

-4.26 

2.87 

90 

-3.09 

3.03 

182 

-3.68 

3.00 

Medium  Difficulty 

96 

-.95 

2.97 

96 

.38 

3.02 

192 

-.67 

3.00 

High  Difficulty 

95 

2.27 

4.14 

92 

3.02 

3.99 

187 

2.64 

4.07 

Total 

283 

-.94 

4.29 

278 

-.13 

4.18 

561 

-.54 

4.25 

Three-Way  Analysis  of  Variance 


Source  of 
Variation 


Sum  of 
Squares 


Mean 

Square 


Main  Effects 

KR  94.27 

Test  Strategy  7.48 

Difficulty  Level  3693.13 

Two-Way  Interactions 

KR  X Test  Strategy  4.82 

KR  X Difficulty  Level  8.96 

Test  Strategy  x 

Difficulty  Level  37.23 

Three-Way  Interaction 
KR  S'  Test  Strategy  x 

Difficulty  Level  63.58 

Residual  6210.13 

Total  10112.87 


1 

94.27 

8.33 

.004 

1 

7.48 

.66 

.999 

2 

1846.56 

163.24 

.001 

1 

4.82 

.43 

.999 

2 

4.48 

.40 

.999 

2 

18.62 

1.65 

.192 

2 

31.79 

2.81 

.059 

549 

11.31 

560 

18.06 

*Estimated  probability  of  error  in  rejecting  null  hypothesis. 


Anxiety.  Table  4 shows  the  means  and  standard  deviaclons  of  the  Anxiety 
scale  scores  as  a function  of  experimental  condition  and  the  results  of  a 
three-way  analysis  of  variance  of  these  scores.  As  Table  4 shows,  there 
were  no  effects  of  KR-provlsion , test  strategy,  or  test  difficulty  on  mean 
Anxiety  scores. 


4 
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Table  4 


Means  and  Standard  Deviations  of  Anxiety  Scores 
for  Conventional  and  Stradaptlve  Tests  With  and  Without  KR 
at  Three  Levels  of  Difficulty,  and  Three-Way  ANOVA  Results 


Experimental 

Condition 

Experimental 

KR 

Condition 

No-KR 

Combined 

Conditions 

N 

Mean 

S.D. 

N 

Mean 

S.D. 

N 

Meein 

S.D, 

Conventional  Test 

Low  Difficulty 

48 

-4.14 

2.64 

45 

-3.70 

2.58 

93 

-3.93 

2.61 

Medium  Difficulty 

47 

-3.49 

3.20 

49 

-3.45 

3.33 

96 

-3.47 

3.25 

High  Difficulty 

46 

-3.34 

2.88 

46 

-3.46 

3.23 

92 

-3.40 

3.04 

Stradaptlve  Test 

Low  Difficulty 

44 

-3.52 

3,09 

45 

-3.84 

3.08 

89 

-3.68 

3.07 

Medium  Difficulty 

49 

-4.06 

2.94 

47 

-4.09 

3.26 

96 

-4.08 

3.08 

High  Difficulty 

49 

-3.72 

3.26 

46 

-3.24 

2.84 

95 

-3.48 

3.06 

Combined  Conditions 

Conventional  Test 

141 

-3.67 

2.91 

140 

-3.53 

3.05 

281 

-3.60 

2.98 

Stradaptlve  Test 

142 

-3.77 

3.08 

138 

-3.73 

3.07 

280 

-3.75 

3.07 

Low  Difficulty 

92 

-3.84 

2.86 

90 

-3.77 

2.83 

182 

-3.81 

2.84 

Medium  Difficulty 

96 

-3.78 

3.06 

96 

-3.77 

3.30 

192 

-3.77 

3,17 

High  Difficulty 

95 

-3.54 

3.07 

92 

-3.35 

3.02 

187 

-3.44 

3.04 

Total 

283 

-3.72 

3.00 

278 

-3.63 

3.06 

561 

-3.67 

3.02 

Three-Way  Analyst 

s of 

Variance 

Source  of 

Sum  of 

Mean 

Variation 

Squares 

DF 

Sauare 

F 

P* 

Main  Effects 

KR 

1.16 

1 

1.16 

.13 

.999 

Test  Strategy 

3,31 

1 

3.31 

.36 

.999 

Difficulty  Level 

15,30 

2 

7.65 

.83 

.999 

Two-Way  Interactions 

KR  X Test  Strategy 

.20 

1 

.20 

.02 

.999 

KR  X Difficulty  Level 

.85 

2 

.42 

.05 

.999 

Test  Strategy  x 

Difficulty  Level 

17.23 

2 

8.61 

.93 

.999 

Three-Way  Interaction 

KR  X Test  Strategy  x 

Difficulty  Level 

10,66 

2 

5.33 

.58 

.999 

Residual 

5067.31 

549 

9.23 

Total 

5115.72 

560 

9.14 

*E3timated  probability  of  error  in  rejecting  null  hypothesis  . 


Motivation.  The  neans  and  standard  deviations  of  the  Motivation  scores 
as  a function  of  experimental  condition  are  indicated  in  Table  5.  A three-way 
analysts  of  variance  of  these  data,  also  shown  In  Table  5,  Indicated  a signifi- 
cant (p<.05)  main  effect  for  the  KR  factor.  The  mean  Motivation  score  for 
individuals  receiving  KR  (.10)  was  significantly  higher  than  that  for 
individuals  not  receiving  KR  (-.32).  No  other  main  effects  or  Interactions  were 
statistically  significant. 
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Table  5 

Means  and  Standard  Deviations  of  Motivation  Scores 
for  Conventional  and  Stradaptive  Tests  With  and  Without  KR 


at  Three  Levels 

of  Difficulty, 

and  Three- 

-Way  ANOVA 

Results 

Experimental 

Condition 

Experimental  Co  dltlon 

KR  No-KR 

Combined 

Condlt  lor.s 

N 

Mean 

S.D. 

R Mean 

S.D. 

N Mean 

S.D. 

Conventionil  Test 

Low  Difficulty 

48 

.08 

2.49 

45  .07 

1.92 

93  .07 

2.22 

Medium  Difficulty 

47 

-.01 

2.31 

49  -.01 

2.33 

96  -.01 

2.31 

High  Difficulty 

46 

.33 

1.96 

46  -.70 

2.14 

92  -.18 

2.11 

Stradaptive  Test 

Low  Difficulty 

44 

.24 

2.34 

45  -.49 

2.10 

89  -.13 

2.24 

Medium  Difficulty 

'.9 

.12 

1.97 

47  -.55 

2.26 

96  -.20 

2.13 

High  Difficulty 

49 

-.16 

2.08 

46  -.25 

2.25 

95  -.21 

2.15 

Combined  Conditions 

Conventional  Test 

141 

.13 

2.26  140  -.21 

2.16 

281  -.04 

2.21 

Stradaptive  Test 

142 

.06 

2.12  138  -.43 

2.19 

280  -.18 

2.17 

Low  Difficulty 

92 

.16 

2.40 

90  -.21 

2.02 

182  -.03 

2.22 

Medium  Difficulty 

96 

.06 

2.13 

96  -.27 

2.30 

192  -.11 

2.22 

High  Difficulty 

95 

.08 

2.03 

92  -.48 

2.19 

187  -.20 

2.12 

Total 

283 

.10 

2.19  278  -.32 

2.17 

531  -.11 

2.19 

Three-Way  Analysis  of  Variance 

Source  of 

Sum  of 

Mean 

Variation 

Squares 

DF 

Square 

F 

P* 

Main  Effects 

KR 

24.37 

1 

24.37 

5.10 

.023 

Test  Strategy 

2.78 

1 

2.78 

.58 

.999 

Difficulty  Level 

2.63 

2 

1.32 

.28 

.995 

Two-Way  Interactions 

KR  X Test  Strategy 

.79 

1 

.79 

.17 

.999 

KR  X Difficulty  Level 

1.38 

2 

.69 

.14 

.999 

Test  Strategy  x 

Difficulty  Level 

.85 

2 

.42 

.09 

.999 

Three-Way  Interaction 

KR  X Test  Strategy  x 

Difficulty  Level 

21.06 

2 

10.53 

2.20 

.999 

Residual 

2623.82 

549 

4.78 

Total 

2677.69 

560 

4.78 

*E3timated  probability  of  error  in  rejecting  null  hypothesis. 


Reaatione  to  KR.  Table  6 shows  the  means  and  standard  deviations  of 
scores  on  the  KR-Reaction  scale  as  a function  of  test  strategy  and  test  diffi- 
culty for  students  in  the  KR  conditions.  Table  6 also  shows  the  results  of  a 
two-way  analysis  of  variance  of  these  data.  There  were  no  significant  effects 
of  experimental  condition  on  mean  KR-Reaction  scores. 
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The  endorsement  frequencies  of  response  options  on  the  KR-Reaction  items 
are  shovm  in  Table  7.  Of  the  283  students  receiving  KR,  87%  said  that  KR 
made  the  test  much  more  Interesting,  86%  felt  that  KR  did  not  interfere 


Table  6 

Means  and  Standard  Deviations  of  KR-Reaction  Scores 
for  Conventional  and  Stradaptlve  Tests  at  Three  Levels 
of  Difficulty,  and  Two-Way  ANOVA  Results 


Experimental 

Condition 

N 

Mean 

S.D. 

Conventional  Test 

Low  Difficulty 

48 

-1.16 

1.62 

Medium  Difficulty 

47 

-.92 

1.59 

High  Difficulty 

46 

-1.26 

1.90 

Stradaptlve  Test 

Low  Difficulty 

44 

-.42 

.83 

Medium  Difficulty 

49 

-.99 

2.12 

High  Difficulty 

49 

-1.  39 

2.30 

Combined  Conditions 

Conventional  Test 

141 

-1.11 

1.70 

Stradaptlve  Test 

142 

-.95 

1.92 

Low  Difficulty 

92 

-.80 

1.35 

Medium  Difficulty 

96 

-.95 

1.87 

High  Difficulty 

95 

-1.32 

2.11 

Total 

283 

-1.03 

1.81 

Two-Way  Analysis  of  Variance 


Source  of 

Sum  of 

Mean 

Variation 

Squares 

DF 

Square 

F 

P* 

Main  Effects 

Test  Strategy 

2.11 

1 

2.11 

.65 

.999 

Difficulty  Level 
Two-Way  Interaction 

13.72 

2 

6.86 

2.11 

.121 

Test  Strategy  x 

Difficulty  Level 

10.84 

2 

5.42 

1.66 

.189 

Residual 

901.68 

in 

3.26 

Total 

928.07 

282 

3.29 

*h!stimated  probability  of  error  in  rejecting  null  hypothesis. 


with  their  ability  to  concentrate  on  the  test,  76%  reported  that  KR  did 
not  make  them  "nervous,"  and  81%  were  very  Interested  in  knowing  whether 
their  answers  were  right  or  wrong.  Ninety-two  percent  of  the  students 
receiving  KR  indicated  that  they  liked  getting  immediate  feedback. 
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Table  7 


Relative  Frequencies  of  Response  Option  Endorsement 
for  KR-Reactlon  Items  (yy-283) 


Item 

Relative  Frequency 
of  Response  Option 
Endorsement 

Did 

getting  feedback  on  this  test  make 

It  more 

interesting  or  less  Interesting? 

1. 

Much  more  interesting 

.87 

2. 

Somewhat  more  interesting 

.11 

3. 

Didn't  make  any  difference 

.00 

4. 

Somewhat  less  Interesting 

.01 

5. 

Much  less  interesting 

.01 

Did 

rece 

ivlng  feedback  after  each  question 

Interfere  with  your  ability  to  concentrate 

on  the  test? 

1. 

No,  not  at  all 

.86 

2. 

Yes , somewhat 

.12 

3. 

Yes,  moderately  so 

.04 

4. 

Yes,  very  much  so 

.02 

Did 

getting  feedback  after  each  question 

make 

you 

nervous? 

1. 

Mo,  not  at  all 

.76 

2. 

Yes,  somewhat 

.22 

3. 

Yes,  moderately  so 

.01 

4. 

Yes , very  much  so 

.01 

Were 

you 

Interested  In  knowing  whether 

your 

answers  were  right  or  wrong? 

1. 

I was  very  Interested 

.81 

2. 

I was  moderately  interested 

.14 

3. 

I was  somewhat  interested 

.04 

4. 

I didn't  care  at  all 

.00 

How 

do  you  feel  about  getting  feedback? 

1. 

I'd  rather  not  know  whether  my 

answers  were  right  or  wrong 

.07 

2. 

I really  don't  care  whether  I 

get  feedback  or  not 

.01 

3. 

1 liked  getting  the  feedback 

.92 

IntervelationahipB  of  peuaholocfiaal  Boales.  The  reliabilities  of  the 
Difficulty-Perception,  Anxiety,  Motivation,  and  K -Reaction  scales  are  shovm 
in  the  diagonal  of  the  matrix  In  Table  8.  The  off-dlagonal  entries  In  the 
matrix  are  the  Pearson  product-moment  correlations  among  the  four  scales. 

The  correlations  Involving  the  KR-Reactlon  scale  are  based  on  the  283  students 
In  the  KR  conditions,  whereas  the  other  correlations  are  based  on  all  561 


students.  As  the  perceived  difficulty  of  the  test  Increased,  Anxiety  scores 
Increased  and  Motivation  scores  decreased.  Anxiety  was  positively  but 


modestly  correlated  with  Motivation.  Reactions  to  KR  became  more  positive  as 
the  Perceived  Difficulty  of  the  test  decreased,  and  Motivation  was  positively 


correlated  with  positive  reactions  to  KR. 


K 

V 
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Table  8 

Reliabilities  and  Intercorrelat Ions  of 
Psychological  Reactions  Scales 

Dlf flculty- 


Scale 

Percept  Ion 

Anxiety 

Motivation  KR-Reactlon 

Difficulty-Perception 

.89 

Anxiety 

.21 

.78 

Motivation 

-.17 

.13 

.62 

KR-Reactlon 

-.15 

-.16 

.25  .66 

Note.  All  correlations 

statistically  significant  at  p^.05. 

PiBOUBsion 

Betz  and  Weiss  (1976a)  found  that  the  provision  of  KR  increased  the 
average  maximum  likelihood  ability  estimate  of  students  administered  a 
stradaptlve  or  a conventional  peaked  test.  They  also  found  that  KR  lncreas€*d 
the  average  number-correct  score  on  the  conventional  teat.  The  present  data 
did  not  replicate  these  findings.  In  this  study  the  average  ability  estimate 
of  students  In  the  KR  and  the  average  of  students  in  the  no-KR  conditions 
did  not  differ  significantly;  nor  did  the  present  data  show  a higher  level 
of  anxiety  on  the  adaptive  test — a finding  reported  In  Betz  and  Weiss  (1976b). 

In  this  study,  the  mean  maximum  likelihood  ability  estimate  was  higher 
on  the  adaptive  test  than  on  the  conventional  test.  In  addition,  the  data 
suggest  that  average  motivation  is  Increased  by  the  provision  of  KR.  This 
latter  effect  may  be  partially  due  to  the  fact  that  students  not  provided 
KR  rated  the  tests  as  more  difficult  than  did  students  receiving  KR.  In 
addition.  Motivation  scale  scores  and  scores  on  the  Difficulty-Perception 
scale  had  a significant  but  modest  negative  correlation. 

Prestwood  and  Weiss  (1977)  analyzed  student  perceptions  of  Individual 
Item  difficulties  and  showed  that  Items  with  difficulties  somewhat  below  the 
ability  levels  of  the  students  were,  on  the  average,  perceived  as  being  "about 
right"  In  difficulty.  They  suggested  that  by  tailoring  tests  so  that  the 
Item  difficulties  are  psychometrlcally  optimal,  adaptive  strategies  may  also 
be  tailoring  tests  so  that.  In  effect,  all  of  the  Items  are  perceived  by 
testees  as  being  too  difficult.  They  concluded  that  the  psychological  effects 
of  such  a procedure  should  be  Investigated  more  fully.  The  data  reported 
here  show  no  significant  effects  on  mean  level  of  anxiety  or  motivation  due  to 
test  difficulty  alone.  These  data  suggest  that  although  students  perceive  Items 
below  their  level  of  ability  as  being  most  appropriate  In  difficulty,  these 
perceptions  do  not  adversely  affect  motivation  when  test  difficulty  Is  In 
the  range  employed  In  this  study  (proportion  correct  of  .40  to  .80). 

The  marginally  significant  but  highly  provocative  Interaction  of  test 
difficulty  and  KR-provlslon  In  the  analysis  of  maximum  likelihood  ability 
estimates  Indicates  that  the  provision  of  KR  may  affect  the  performance  of 

Individuals  dlf ferentlally,  depending  on  the  difficulty  of  the  task.  On  a | 

conventional  test.  Individual  students  may  receive  different  tests  In  a | 

psychological  sense.  Although  each  student  will  receive  the  same  Items,  low-  t 

ability  students  may  be  responding  to  a relatively  "difficult"  test,  whereas  I 
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hlgh-abll Ity  students  may  be  responding  to  a relatively  "easy"  test.  If  the 
provision  of  KR  has  differential  effects  depending  on  the  difficulty  of  the 
task,  students  of  different  ability  levels  may  respond  differently  to  the 
provision  of  KR  according  to  their  perceptions  of  the  task's  difficulty,  even 
though  tliey  are  administered  the  same  set  of  test  items. 

On  an  adaptive  test,  where  the  difficulty  of  the  task* is  modified  so  that 
each  individual  answers  approximately  the  same  proportion  of  items  correctly, 
the  differential  effects  of  KR  and  test  difficulty  should  be  minimized. 
Although  students  react  very  favorably  to  KR  regardless  of  the  proportion  of 
positive  feedback  tfiey  received,  the  interaction  of  test  difficulty  and  KR- 
provislon  in  the  analysis  of  ability  estimates  suggests  that  the  effects  of 
KR's  provision  on  perfof.  moe  should  be  carefully  investigated  before  KR  is 
provided  under  new  sets  of  conditions. 


I 

1 

1 


I 


I 
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Appendix:  Supplementarn  Tables 


Table  A 

Difficulty-Perception  Items,  Serial  Position  of  Administration, 
and  Weights  Assigned  to  Response  Alternatives 


Item  and  Response 

Alternatives 

Serial 

Position 

Assigned 

Weight 

How  o 

ften  did  you  feel  that  the  questions  in 

the  test  were  too  easy  for  you? 

1 

1, 

Always 

1.00 

2. 

Frequently 

2.00 

3. 

Sometimes 

3.00 

4. 

Seldom 

4.00 

5. 

Never 

5.00 

How  often  did  you  feel  that  the  questions  in 

the  test  were  too  hard  for  you? 

2 

1. 

Always 

-1.00 

2. 

Frequently 

-2.00 

3. 

Sometimes 

-3.00 

4. 

Seldom 

-4.00 

5. 

Never 

-5.00 

On  how  many  of  the  questions  did  you  guess? 

5 

1. 

Almost  all  of  the  questions 

-1.00 

2, 

More  than  half  of  the  questions 

-1.80 

3. 

About  half  of  the  questions 

-2.60 

4. 

Less  than  half  of  the  questions 

-3.40 

5. 

Almost  none  of  the  questions 

-4.20 

6. 

None  of  the  questions 

-5.00 

How  often  were  you  sure  that  your  answers  to  the 

questions  were  correct? 

8 

1. 

Almost  always 

1.00 

2. 

More  than  half  of  the  time 

2.00 

3. 

About  half  of  the  time 

3.00 

4. 

Less  than  half  of  the  time 

4.00 

5. 

Almost  never 

5.00 

In  relation  to  your  vocabulary  ability,  how 

difficult  was  the  test  for  you? 

10 

1. 

Much  too  difficult 

-1.00 

2. 

Somewhat  too  difficult 

-2.00 

3. 

Just  about  right 

-3.00 

4. 

Somewhat  too  easy 

-4.00 

5. 

Much  too  easy 

-5.00 

Did  you  feel  frustrated  by  the  difficulty  of 

the  test  questions? 

1. 

Not  at  all 

12 

1.00 

2. 

Somewhat 

2.33 

3. 

Fairly  much  so 

3.67 

4. 

Very  much  so 

5.00 
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Table  B 

Anxiety  Items,  Serial  Position  of  Administration, 
and  Weights  Assigned  to  Response  Alternatives 


Item  and  Response 

Alternatives 

Serial 

Position 

Assigned 

Weight 

During 

testing,  did  you  worry  about  how  well 

you  would  do? 

4 

1. 

Not  at  all 

1.00 

2. 

Somewhat 

2.33 

3. 

Fairly  much  so 

3.67 

4. 

Very  much 

5.00 

Were  you  nervous  while  taking  the  test? 

7 

1. 

Not  at  all 

1.00 

2. 

Somewhat 

2.33 

3. 

Moderately  so 

3.67 

4. 

Very  much  so 

5.00 

How  did  you  feel  while  taking  the  test? 

11 

1. 

Very  tense 

-1.00 

2. 

Somewhat  tense 

-2.00 

3. 

Neither  tense  nor  relaxed 

-3.00 

4. 

Somewhat  relaxed 

-4.00 

5. 

Very  relaxed 

-5.00 

Did  nervousness  while  taking  the  test  prevent 

you  from  doing  your  best? 

16 

1. 

Yes,  definitely 

-1.00 

2. 

Yes,  somewhat 

-2.33 

3. 

Probably  not 

-3.67 

4. 

Definitely  not 

-5.00 

Table  C 

Motivation  Items,  Serial  Position  of 
and  Weights  Assigned  to  Response 

Administration, 

Alternatives 

Item 

and  Response 

Serial 

Assigned 

Alternat Ives 

Position 

Weight 

How  frequently  were  you 

careful  to  select  what 

you 

thought  was  the  best  answer  to 

each  question? 

6 

1 . 

Almost  always 

-1.00 

2. 

Frequently 

-2.00 

3. 

Somct Imes 

-3.00 

4. 

Rarely 

-4.00 

5. 

Never 

-5.00 

Did  you  feel  challenged 

to  do 

as  well  as  you 

could 

on  the  test? 

13 

1. 

Not  at  all 

1.00 

2. 

Somewhat 

2.33 

3. 

Fairly  much  so 

3.67 

4. 

Very  much  so 

5.00 

Did  you  care  how  well  you  did 

on  the  test? 

18 

1. 

I cared  a lot 

-1.00 

2. 

I cared  some 

-2.00 

3. 

1 cared  a little 

-3.00 

4. 

I cared  very  little 

-4.00 

5. 

I didn't  care  at 

all 

-5.00 

D r-— -m r-wrrn^mmmmr* 
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Table  D 

KR-Reactlon  Items,  Serial  Position  of  Administration, 
and  Weights  Assigned  to  Response  Alternatives 


Item  and  Response 

Alternatives 

Serial 

Posit  ion 

Assigned 

Weight 

Did  getting  feedback  on  this  test  make  It 

more  Interesting  or  less  Interesting’ 

19 

1.  Much  more  Interesting 

-1.00 

2.  Somewhat  more  Interesting 

-1.75 

3.  Didn't  make  any  difference 

-2.50 

4.  Somewhat  less  Interesting 

-3.25 

5.  Much  less  Interesting 

-4.00 

Did  receiving  feedback  after  each  question 

Interfere  with  your  ability  to  concentrate 

on  the  teat? 

20 

1.  No,  not  at  all 

-1.00 

2.  Yes,  somewhat 

-2.00 

3.  Yes,  moderately  so 

-3.00 

4.  Yes,  very  much  so 

-4.00 

Did  getting  feedback  after  each  question 

make  you  nervous? 

21 

1.  No,  not  at  all 

-1.00 

2.  Yes,  somewhat 

-2.00 

3.  Yes,  moderately  so 

-3.00 

4.  Yes,  very  much  so 

-4.00 

Were  you  Interested  In  knowing  whether  your  answers 

were  right  or  wrong? 

24 

1.  I was  very  Interested 

-1.00 

2.  I was  moderately  interested 

-2.00 

3.  I was  somewhat  interested 

-3.00 

4.  I did  not  care  at  all 

-4.00 

How  do  you  feel  aboit  getting  feedback? 

26 

1.  I'd  rather  not  know  whether  my  answers 

were  right  or  wrong 

1.00 

2.  I really  don't  care  whether  I get  feedback 

or  not 

2.50 

3.  I liked  getting  tfie  feedback 

4.00 
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Table  E 

Normal-Ogive  Discrimination  (a)  and  Difficulty  (b) 
Parameters  of  Items  on  the  Three  50-Item  Conventional  Tests 


Low-Difficulty 

Teat 

Med ium-Dif  f icul ty 

Test 

Hlgh-Dif f Iculty  Test 

Item 

Reference 

Number 

a 

b 

Item 

Reference 

Number 

a 

b 

Item 

Reference 

Number  a 

b . 

20 

.38 

-5.76 

23 

.71 

-3.86 

262 

.77 

-1.93 

72 

.27 

-6.13 

18 

.48 

-4.24 

31 

.72 

-2.14 

4 

.40 

-5.56 

240 

.59 

-3.35 

641 

.58 

-1.40 

29 

.32 

-5.52 

212 

.56 

-3.64 

559 

.62 

-1.68 

62 

.43 

-4.95 

16 

.75 

-2.95 

285 

.84 

-1.02 

78 

.44 

-4.84 

100 

.59 

-3.18 

640 

.78 

-1.06 

55 

.29 

-4.95 

201 

.31 

-2.97 

26 

.36 

-1.02 

23 

.71 

-3.86 

65 

1.02 

-2.71 

43 

1.11 

-.86 

18 

.48 

-4.24 

89 

.72 

-2.49 

149 

.83 

-.46 

122 

1.15 

-3.31 

124 

1.09 

-2.64 

87 

1.24 

-.76 

32 

.66 

-3.73 

63 

.69 

-2.14 

33 

.80 

-.39 

2 

.52 

-3.81 

131 

.60 

-2.58 

183 

.73 

-.45 

212 

.56 

-3.64 

106 

.67 

-2.01 

207 

. 79 

-.04 

77 

.44 

-3.60 

81 

.44 

-2.39 

157 

.38 

-.25 

8 

1.01 

-3.15 

186 

1.07 

-1.34 

599 

1.63 

.16 

39 

.35 

-3.63 

256 

. 46 

-1.93 

205 

.60 

-.02 

19 

.71 

-3.81 

559 

.62 

-1.68 

144 

.91 

.29 

182 

.70 

-3.83 

34 

.83 

-1.58 

568 

1.63 

.29 

105 

.98 

-2.63 

127 

1.08 

-1.35 

283 

3.00 

.49 

261 

.42 

-3.46 

276 

.45 

-1.53 

318 

.53 

.31 

16 

.75 

-2.95 

285 

.84 

-1.02 

340 

1.92 

.65 

187 

.48 

-3.07 

94 

.56 

-1.02 

506 

.81 

.58 

121 

.74 

-2.82 

85 

.93 

-.67 

1J4 

3.00 

.96 

69 

.53 

-3.23 

241 

.57 

-1.05 

271 

.89 

.80 

126 

.96 

-2.27 

522 

1.06 

-.39 

541 

3.00 

1.16 

131 

.60 

-2.58 

222 

.65 

-.50 

526 

1.17 

.92 

66 

.87 

-2.02 

270 

1.22 

-.14 

665 

3.00 

1.62 

63 

.69 

-2.14 

156 

.84 

-.17 

321 

3.00 

1.00 

262 

.77 

-1.93 

56 

1.11 

.14 

120 

3.00 

1.46 

95 

.56 

-1.71 

444 

.62 

.06 

306 

1.32 

1.20 

90 

.94 

-1.31 

568 

1.63 

.29 

174 

3.00 

1.46 

186 

1.07 

-1.34 

266 

2.12 

.51 

254 

3.00 

1.37 

276 

.45 

-1.53 

629 

.53 

.42 

147 

.83 

1.47 

214 

.48 

-1.49 

377 

.59 

.39 

586 

1.54 

1.31 

322 

.67 

-1.09 

113 

1.06 

.68 

652 

3.00 

1.66 

235 

.66 

-.78 

271 

.89 

.80 

595 

3.00 

1.58 

349 

.92 

-.52 

114 

3.00 

.96 

573 

3.00 

1.86 

123 

.82 

-.56 

294 

3.00 

1.07 

319 

3.00 

2.14 

287 

.52 

-.65 

111 

.62 

.94 

400 

.93 

1.68 

94 

.56 

-1.02 

506 

.81 

.58 

263 

3.00 

1.47 

234 

.65 

-.13 

601 

1.32 

1.10 

445 

3.00 

2.07 

597 

.62 

-0 

299 

1.77 

1.16 

383 

2.11 

1.52 

145 

.79 

.09 

581 

1.26 

1.21 

247 

.65 

2.06 

292 

.61 

.01 

306 

1.32 

1.20 

273 

3.00 

2.14 

635 

.57 

.27 

304 

.89 

1.34 

115 

3.00 

2.02 

622 

.44 

.20 

367 

.72 

1.40 

533 

.63 

2.15 

506 

.81 

.58 

140 

3.00 

1.38 

381 

3.00 

2.35 

165 

.38 

.56 

286 

.59 

1.64 

534 

2.52 

1.61 

113 

1.06 

.68 

595 

3.00 

1.58 

360 

3.00 

1.71 

526 

1.17 

.92 

573 

3.00 

1.86 

609 

3.00 

"Heari” 

S.D. 

.23 

"-T.IT 

1.91 

Mean 

S.D. 

i.of 

.73 

-.70 

1.71 

Mean 

S.D. 

1.76 

1.05 

.68 

1.21 
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Table  F 

Normal-Ogive  Discrimination  (a)  and  Difficulty  (b)  Parameters 
for  Items  in  the  Stratified  Item  Pool 

Item  Item  Item 

Reference  Reference  Reference 

Number  a b Number a b Number a b 


Stratum  1 


573 

3.00 

1.86 

595 

3,00 

1.58 

263 

3.00 

1.47 

378 

3.00 

1.48 

561 

1.72 

1.42 

572 

1.29 

1.43 

253 

2.32 

1.44 

168 

.91 

1.55 

260 

.71 

1.82 

400 

.93 

1.68 

521 

.75 

1.70 

504 

.64 

1.81 

616 

.61 

1.76 

403 

.54 

1.76 

577 

.61 

2.00 

374 

.56 

1.99 

167 

.42 

2.16 

Mean 

1.41 

1.70 

S.D. 

.99 

.22 

Stratum 

2 

665 

3.00 

1.62 

120 

3.00 

1.46 

254 

3.00 

1.37 

288 

3.00 

1.26 

562 

3.00 

1.22 

617 

2.78 

1.17 

299 

1.77 

1.16 

140 

3.00 

1.38 

306 

1.32 

1.20 

581 

1.26 

1.21 

291 

1.64 

1.35 

217 

1.25 

1.38 

304 

.89 

1.34 

587 

.87 

1.36 

147 

.83 

1.47 

367 

.72 

1.40 

216 

.67 

1.40 

397 

.65 

1.34 

610 

.79 

1.57 

159 

.77 

1.56 

107 

.69 

1.59 

525 

.57 

1.51 

286 

.59 

1.64 

242 

.52 

1.57 

505 

.50 

1.43 

Stratum 

2 

(cont 'd) 

119 

.53 

1.73 

281 

.40 

1.79 

603 

.38 

1.80 

Mean 

1.37 

1.44 

S.D. 

.98 

.18 

Stratum 

3 

583 

3.00 

1.16 

337 

3.00 

1.18 

541 

3.00 

1.16 

114 

3.00 

.96 

294 

3.00 

1.07 

321 

3.00 

1.00 

607 

3.00 

1.11 

526 

1.17 

.92 

651 

1.09 

.89 

601 

1.32 

1.10 

598 

1.08 

1.04 

111 

.82 

.94 

375 

.83 

.93 

215 

.91 

1.07 

231 

.87 

1.19 

238 

.76 

1.13 

164 

.69 

1.14 

341 

.63 

1.28 

368 

.46 

1.42 

576 

.43 

1.13 

213 

.43 

1.43 

259 

.37 

1.29 

172 

.38 

1.36 

516 

.35 

1.12 

333 

.35 

1.34 

308 

.34 

1.31 

Mean 

1.32 

1.14 

S.D. 

1.05 

.15 

Stratum 

4 

347 

3.00 

.49 

296 

3.l3 

.67 

266 

2.12 

.51 

264 

2.28 

.55 

340 

1.92 

.65 

342 

1.59 

.54 

265 

1.57 

.55 

301 

1.38 

.47 

582 

1.20 

.35 

(continued!) 


Stratum 

4 

(cont'd) 

386 

1.25 

.54 

538 

1.18 

.52 

60 

1.23 

.64 

59 

1.09 

.60 

113 

1.06 

.68 

551 

.90 

.34 

146 

.93 

.47 

636 

.76 

.37 

271 

.89 

.80 

506 

.81 

.58 

295 

.68 

.53 

233 

.65 

.41 

267 

.65 

.77 

139 

.61 

.79 

133 

.57 

.56 

258 

.57 

.64 

593 

.56 

.55 

519 

.53 

.44 

289 

.48 

.69 

549 

.43 

.35 

252 

.42 

.47 

Mean 

1.14 

.55 

S.D. 

.70 

.13 

Stratum 

5 

380 

1.82 

.12 

272 

1.96 

.22 

599 

1.63 

.16 

568 

1.63 

.29 

329 

1.42 

.18 

161 

1.38 

.13 

270 

1.22 

-.14 

128 

1.07 

-.36 

143 

1.04 

-.15 

37 

.86 

-.24 

156 

.84 

-.17 

221 

.82 

-.28 

209 

.87 

.07 

144 

.91 

.29 

211 

.77 

-.24 

52 

.84 

.21 

207 

.79 

-.04 

208 

.74 

-.18 

369 

.79 

.30 

307 

.70 

-.33 

22L- 

.68 
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Table  F {aont  'd) 


Item 

Reference 

Number 

a 

h 

Item 

Reference 

Number 

a 

b 

Item 

Reference 

Number 

a 

b 

Stratum 

5 

(cont' 

d) 

Stratum 

7 

(cont ' d) 

Stratum 

8 

(cont ' 

'd) 

502 

.73 

.22 

190 

1.82 

-1.44 

80 

.86 

-2.25 

50 

.69 

.32 

27 

1.43 

-1.68 

198 

.80 

-2.50 

391 

.62 

.06 

84 

1.70 

-1.64 

5 

.75 

-2.16 

292 

.61 

.01 

13 

1.89 

-1.55 

89 

.72 

-2.49 

205 

.60 

-.02 

96 

1.13 

-1.72 

184 

.73 

-2.19 

355 

.51 

.10 

125 

1.24 

-1.88 

31 

.72 

-2.14 

218 

.41 

-.13 

129 

1.27 

-1.35 

63 

.69 

-2.14 

23A 

.65 

-.13 

22 

1.20 

-1.97 

106 

.67 

-2.01 

157 

.38 

-.25 

101 

1.17 

-1.40 

255 

.64 

-2.18 

Mean 

.93 

-.01 

44 

1.15 

-1.41 

202 

.62 

-2.17 

S.D. 

.40 

.21 

134 

1.07 

-1.94 

131 

.60 

-2.58 

Stratum 

6 

127 

1.08 

-1.35 

628 

.57 

-2.29 

19A 

1.79 

-.96 

186 

1.07 

-1.34 

82 

.54 

-2.31 

36 

1.64 

-.  79 

90 

.94 

-1.31 

151 

.44 

-2.65 

40 

1.24 

-1.03 

83 

.88 

-1.45 

73 

.43 

-2.69 

87 

1.24 

-.  76 

86 

.89 

-1.19 

Mean 

1.05 

-2.34 

501 

1.20 

-.55 

34 

.83 

-1.58 

S.D. 

.62 

.23 

199 

43 

109 

47 

522 

239 

173 

85 

1.09 

1.11 

1.11 

1.04 

1.06 

.94 

.88 

.93 

-1.09 

-.36 

-.70 

-.96 

-.39 

-.71 

-1.06 

-.67 

227 

262 

311 

189 

88 

232 

76 

95 

.81 

.77 

.75 

.76 

.71 

.67 

.62 

.56 

-1.25 

-1.93 

-1.43 

-1.19 

-1.33 

-1.25 

-1.75 

-1.71 

Stratum 
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